Updated, I've overlooked the else branch for empty scap contents :/

Test failures are not related, but look pretty scary :(

UPD: Weird, we have github actions, which run the same? set of tests and it passes, but jenkins does not...

Because there are github actions-based tests (which are green), I assumed the jenkins tests are a remnant of the past which we just failed to disable, but I could be wrong about that.

Should we remove jenkins tests then? I'm not so familiar with this plugin and not sure we can rely on GA only...

Per #473 (comment) I would say yes

I've opened PR to remove jenkins job: theforeman/jenkins-jobs#360

Just for context, this is actually an issue we see in more plugins. There's a difference in how things run.

In GHA there is just a single "install" step. For Jenkins we first run Foreman, install & migrate (db:migrate & db:seed). Then the plugin is added and it's installed & migrated. Then Foreman also runs the entire test suite where I think GHA only runs the plugin specific tests.

The idea behind Jenkins is that it does regression testing and also testing the scenario where it's added to an existing Foreman. Whether it actually achieves those 2 things is a good question.

@stejskalleos and I also talked about this in the past, but I don't think we ever resolved it. I'd appreciate it if we looked at it from a broader project wide perspective.

Just for context, this is actually an issue we see in more plugins. There's a difference in how things run.

In GHA there is just a single "install" step. For Jenkins we first run Foreman, install & migrate (db:migrate & db:seed). Then the plugin is added and it's installed & migrated. Then Foreman also runs the entire test suite where I think GHA only runs the plugin specific tests.

The idea behind Jenkins is that it does regression testing and also testing the scenario where it's added to an existing Foreman. Whether it actually achieves those 2 things is a good question.

@stejskalleos and I also talked about this in the past, but I don't think we ever resolved it. I'd appreciate it if we looked at it from a broader project wide perspective.

I'll try to help, but no promises :/ Where should I start looking? I mean, it's obvious to look at the jenkin's console output for the run, but I haven't found anything useful, since I can't reproduce it locally unless (?) I'll create a new VM with the same steps jenkins has, but I guess nobody wants to spend time on that...

I mean, mostly I'd like to get rid of red status for jenkins here... And why it's not possible to do the same steps via GA?

I mean, mostly I'd like to get rid of red status for jenkins here... And why it's not possible to do the same steps via GA?

It is, but I'd like it if various plugins had some standard way of running CI. GHA is awful in the amount of duplication (by default) so I started https://github.com/theforeman/actions#foreman-plugin-tests. Never got around to completing it.

Then plugins that require Katello had a problem that they needed qpid headers, which weren't available on the Ubuntu based images. That's now gone, so it probably matters less.

AFAIK foreman_openscap doesn't require openscap itself, that's just smart_proxy_openscap. Otherwise you would also need to get libopenscap on Debian. Though I did see that Debian 12 has added libopenscap25.

Those are reasons why Jenkins still gives you better control.

As for the specific failures: I haven't tried to reproduce it myself yet. I think these would be roughly the steps:

git clone https://github.com/theforeman/foreman
cd foreman
# Make sure config/settings.yaml and config/database.yml are correct
bundle install
bundle exec rake db:migrate
bundle exec rake db:seed
cat > bundler.d/openscap.local.rb <<EOF
gem 'foreman_openscap', github: 'theforeman/foreman_openscap'
EOF
bundle install
bundle exec rake db:migrate
bundle exec rake db:seed

I've created an issue ticket to track Jenkins related issue: https://projects.theforeman.org/issues/36804, so we can go ahead with this PR, since it's more visual fix, not something that may introduce regressions.

@adamruzicka, could you please re-take a look? 👼

Thank you @ofedoren !